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Abstract 

Introduction. Reports findings from research in progress investigating 
judgment and decision making during information seeking in the workplace, 
in particular, the assessment of enough information. Characteristics of this 
judgment and the role of context in shaping it are framed against theories of 
human judgment and decision making. 

Method. Thirty-three semi-structured interviews were conducted with public 
sector policy workers in Australia. Two interviews were carried out, the first 
with individual participants and the second, a joint interview with two 
participants. Interviews were taped and transcribed and inductive data 
analysis carried out. 

Findings. Findings discussed in this paper focus on contextual factors that 
frame policy workers' judgment and decision making while information 
seeking, factors including ill-structured problems, shifting goals, time stress 
and action-feedback loops. Also revealed was the importance of developing 
a framework, against which the judgment of enough information can be 
made, and the fluid and iterative nature of these judgments. 

Conclusion. The contextual factors reported show similarities with those 
identified by naturalistic decision making researchers, suggesting this new 
field of decision theory has much to offer researchers into information 
seeking in context. 

Introduction 

This paper reports on ongoing research aimed at developing a deeper 
understanding of the concept of enough information during information seeking. 
Although often depicted as a linear sequence, researchers (e.g., Kuhlthau 2004 : 
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Wilson, et al. 2002 ) acknowledge that information seeking is not experienced as 
a neat, single-pass process and recently others (e.g., Foster 2004 f have developed 
non-linear models to emphasise this attribute. Search closure or, to use Foster's 
terminology, consolidation, has received little focused research attention. While 
some common themes have emerged in recent empirical studies, there is much we 
still need to understand about the nature of the phenomenon of enough 
information and its relationship to what some researchers term the stopping rules 
which govern to decision to stop seeking further information. 

Several themes related to the experience of decision making during information 
seeking emerged during preliminary data analysis, themes that included the 
relationship between the deadline and the assessment of enough information and 
the fluid and iterative way in which that assessment is made. Implications are 
discussed for an expanded understanding of how judgements and decisions are 
made during information seeking. Similarities with decisions studied in the field 
of naturalistic decision making are examined and the suggestion is made that this 
field of decision theory offers a new perspective from which to investigate 
judgements and decisions during information seeking. 

Framing the research - judgements and decisions during search 

closure 

Framed by a theoretical perspective situating information behaviour in context, 
the research conceptualised human information behaviour in a series of nested 
relationships ( Wilson 1999 ). The focus of the research was information seeking 
behaviour, the purposive looking for information to meet a need, although it has 
drawn on findings from a broad range of human information behaviour literature. 
While judgements are made throughout information seeking, this study focused 
on the assessment of enough information made during the closing stages of 
information seeking, as a precursor to putting together a finished written work 
product. 

Understanding that people face a series of choices throughout the information 
search process f Kuhlthau 2004 1 suggested that the field of human judgement and 
decision making would offer insights into the relationship between enough 
information and information seeking and stopping. The classical definition of a 
decision is choosing between two options ( Hardman and Macchi 2003 1. with the 
acts of judging the situation and making a decision seen as 'parts of a multistage 
cognitive process' ( Jungermann 2000 : 587) and the decision itself a commitment 
to action following ongoing evaluation of options ( Harrison 1999 : 5). The 
following section frames findings from human information behaviour research 
which illuminate judgements and decisions made during information seeking 
against the major theories of human judgement and decision making. 

Research into stopping behaviour 

Stopping behaviour, that is, user termination of a search for information, was 
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investigated during the 1970s and 1980s in theoretical and experimental studies, 
often with a particular focus on information retrieval from databases and usually 
framed within theories of rational or classical decision making (e.g., Kantor 1987 : 
Kraft and Lee 1979 : Kraft and Waller 1981 : Morehead and Rouse 1982 ) Within 
this classical decision making tradition, theorists developed normative models 
which prescribed optimal decision making behaviour ( Kerstholt and Avton 2001 
)• 

Researchers sought to determine rules for the 'optimal stopping point' of an 
information retrieval search and investigated how stopping rules of satiation, 
disgust or a combination of the two affected that search length ( Kraft and Waller 
1981 : 349). Other stopping rules investigated included time constraints, 
diminishing returns and frustration t Morehead and Rouse 1982 1. Using Bayesian 
probability theory, Kantor demonstrated 'a very simple cutoff criterion' to end the 
search ( 1987 : 211) employing both the searcher's prediction of success and their 
confidence in that prediction. 

While these studies offered insights into the decision to stop searching for further 
information, the laboratory-based experiments and mathematic modelling were 
far removed from the real world context in which humans make judgements and 
decisions during search closure thus limiting their applicability to real world 
situations ( Case 2002 ). As well, many of the studies focused solely on 
information retrieval and were based on a closed decision model. 

The work of the classical decision theorists generated a reaction as researchers 
broadly grouped within the behavioural school of decision making recognised and 
studied where and how people deviated from the optimal decision making model 
( Newell 2005 ). Particularly influential were Simon's theories of bounded 
rationality. Although he theorised about sequential decision making, Simon's 
( 1997 ) ideas on the inter-related constructs of satisficing and levels of aspiration 
provided insights into judgement and decision making during information 
seeking. Satisficing sees an individual accepting an alternative good enough to 
allow them to achieve the outcome they seek rather than seeking the optimal 
outcome. In an information context, satisficing suggests individuals stop 
searching at that point, even while possibly recognising that further searching 
may well yield additional or even better information. Importantly, satisficing 
occurs against pre-existing levels of aspiration, shaped by personal experience 
and contextual factors ( March 1994 : Simon 1997 1. 

Two recent studies by Agosto and Zach have drawn explicitly on Simon's theories 
of bounded rationality and satisficing to understand the choices being made 
during information seeking. Agosto reported the anticipated time and cognitive 
constraints operating during web-based searching and grouped 'satisficing 
behaviours' ( Agosto 2001 : 23) into two: reduction methods to limit and organize 
the amount of information being dealt with, and termination methods which 
included satisficing (stopping when an 'acceptable' site was found, physical 
discomfort, boredom, self-imposed time limits) and 'snowballing' (that is 
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'repetition' or redundancy of information). Zach also reported time as an issue 
although she positioned time - together with a sense of'comfort' ( Zach 2005 : 31) 
with the amount of information found - as a 'stopping rule'. 

Investigating assessments of enough information 

Other human information behaviour researchers have illuminated the experience 
of search/seeking closure without explicitly drawing on human judgement and 
decision making theories. Time available, positioned as one of the major 
constraints by behavioural decision theorists, is a recurrent theme (e.g., Kuhlthau 
2004 : Limberg 1999 ) 'Task, interest, information available and time' ( Kuhlthau 
2004 : 101) all play a part in influencing the decision to end information seeking 
activity and, in Kuhlthau's studies of the information seeking behaviour of 
professionals, the context in which the work product was to be used was a factor 
in determining enough information. Additionally, increasing certainty signals 
search closure, certainty that the most important issues have been addressed 
( Kuhlthau 2004 : Wilson, et al. 2002 ) and a sense that the puzzle is looking 
completed ( Kuhlthau 2004 ). Increased certainty was evident too in Zach's arts 
administrators' 'arbitrary level of comfort' with the amount of information they 
had gathered and the sense of building a complete picture found parallels in 
empirical work by Cole ( 1997 ) and Foster (2004). 

As well as time constraints and confidence, a personal investment in the quality of 
the assignment being completed was a further way in which students described 
their experiences of assessing enough information ( Limberg 1999 : Parker 2004 ). 

This review reveals attempts to understand search closure have approached the 
phenomenon from various research perspectives and investigated different aspects 
of the phenomenon. Major themes running through empirical findings about the 
closing stages of information seeking included time and cognitive limitations in 
the form of deadlines and information overload; increasing certainty and 
confidence in being able to present a case; increasing redundancy in the 
information located; and a sense of having put together a complete picture. 
Framing the information behaviour research findings to date against theories of 
human judgement and decision making, we can differentiate between the action 
(the decision to stop or continue) and the evaluations (including the judgement of 
enough information) that precede that action. 

The study 

To examine some of the questions arising out of the findings reported above, this 
study sought to understand, first, how people in the workplace made an 
assessment they have enough information and secondly, what influences were 
apparent that might shape those assessments. 

Study participants 
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Public sector policy workers of state government organizations in Australia were 
the participants in this research. This group and setting were selected for sound 
methodological reasons, as they provided a site where the phenomenon being 
studied could be found in abundance and where the setting was familiar ( Morse 
and Richards 2002 : Patton 2002 1. 

Public sector workers operate in a volatile environment - in which the ambiguous 
and complex nature of policy work I Considine 1994 : Gualtieri 1999 1 is 
characterised by 'continuous work on persistent issues' ( Considine 1994 : 189) in a 
process which is convoluted and unpredictable. Policy workers are players in this 
process, seeking and using information throughout, both as a resource (or input) 
and as an output ( Kirk 1999 ). 

Diversity was apparent in both length of participants' experience in policy (from 
nine months to thirty years), and in the level at which they worked (from a recent 
first appointment to a senior executive). Diversity was also apparent in the nature 
of the work tasks in which their information seeking was embedded. These tasks 
ranged from the frequent and routine preparation of briefing papers for 
Ministerial meetings, albeit often on topics of which the policy workers had little 
existing knowledge, to projects described as once-in-a-career experiences such as 
the development of major ground-breaking legislation. Although some of the 
tasks were described by participants as routine, none fell into the categories of 
'automatic' or 'normal' information processing tasks ( By strom and Jarvelin 1995 : 
194). Rather the tasks, characterised often by their unexpected and unstructured 
nature, appeared to have more in common with what Bystrom and Jarvelin 
identify as 'genuine decision tasks' ( 1995 : 195), even though the policy workers 
themselves were not decision makers. 

Research Design 

Interpreting and understanding a social phenomenon - in this case, how individual 
workers assess they have enough information and decide to stop seeking more - 
required an approach that facilitated an in-depth analysis of the context and the 
people under investigation and mandated a research approach situated within the 
naturalistic paradigm ( Lincoln and Guba 1985 1. In keeping with this naturalistic 
posture, qualitative data collection methods were used ( Morse and Richards 
2002 1. The research was designed as a multiple case study, an approach 
appropriate for exploratory, descriptive and interpretive studies ( Lincoln and 
Guba 1985 : Yin 2002 1. A case may be an individual or an organization, a 
program or some 'event or entity [including] decisions' ( Yin 2002 : 23); in this 
study, the case was the judgement of enough information. 

Given that most workers do not see information seeking as a discrete and 
immediately identifiable activity ( Kuhlthau 2004 : Solomon 1997 1. it was 
important to develop a research design and techniques that would help overcome 
the 'hidden' nature of information seeking at work. As well, the exploratory nature 
of the research suggested a holistic approach was important ( Yin 2002 1. that is, an 
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approach that would allow participants to bring to the surface those factors 
important to them when assessing whether they had enough information. Much 
research into information seeking behaviour has focused on a small number of 
factors identified before the investigation. However, variables that, potentially, 
may influence information seeking behaviour exist on several dimensions ( Fidel 
and Pejtersen 2004 ). suggesting a focus on just two or three may constrain a 
researcher's understanding of the phenomenon, or worse, mislead ( Fidel and 
Pejtersen 2004 T A further compelling reason for a holistic approach is that this is 
how individuals themselves view the activity of information seeking ( Kuhlthau 
2004 ). 

Data Collection and Analysis 

Twenty one policy workers participated in the study, in a total of thirty three 
interviews. In all, two semi-structured interviews were conducted with each 
participant. The first interview was with the individual participant, focusing on 
their role and nature of the work being done. After Fidel and Green ( 2004 ). at the 
conclusion of the first interview, participants were asked to recall and describe a 
particular piece of work during which they had carried out information seeking 
activity. Also, the purpose of the second interview was explained and questions 
for the second interview asked, without answers being sought. The second 
interview was conducted with participants in pairs and focused on the closing 
stages of the information seeking experience. Joint interviews provided a way to 
increase the richness of the data gathered, with the sharing of information 
expected to trigger thoughts and ideas about information behaviour for both 
participants. These features of the research design were designed to assist 
participants in reconstructing the information seeking incident ( Fidel and Green 
2004 ) and overcoming the 'hidden' nature of information seeking at work. 

Interviews were taped and transcribed. Analysis began when the tapes from the 
first interviews were listened to before the second ones took place. Following this 
first step in analysis, as the tapes were being transcribed, a fuller thematic 
analysis was undertaken., in the form of reading the transcripts and listening to 
the tapes to understand the concepts being discussed and identify salient concepts 
and their inter-relationships ( Ezzv 2002 ). A comprehensive case analysis (Patton 
2002 ) is still being carried out. However, the following discussion has been drawn 
from the initial thematic analysis. 

Policy workers' experiences of information seeking and stopping: 

emerging themes and concepts 

As anticipated and foreshadowed in the earlier discussion of the research design, 
participants found it extremely difficult to distinguish the information seeking 
subtask from the broader work task ( Bvstrom and Hansen 2005 1. This confirmed 
findings from a number of earlier empirical studies (e.g. Kuhlthau 2004 : Solomon 
1997 1 and supported Vakkari's ( 2002 : 433) findings that information seeking is 
'systematically connected' to the task being carried out. Because participants often 
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spoke of the two interchangeably, the following discussion only differentiates 
task from information seeking activity when participants did so. 

Deadlines less important 

Not surprisingly, the influence of time as a constraint was supported, with 
participants reporting that time constraints in the form of deadlines were very 
important in the decision to stop searching for more information. Deadlines drove 
these participants and caused them great stress: 7 was beside myself, 7 wanted to 

leave the country' (8) - ; the work is 'nerve-wracking [because] the timelines are 
so short now' (3). However, in spite of the importance of the deadlines, they were 
seen as flexible and negotiable, and often could be extended if necessary: 'But 
also, I think surprisingly even though you do have a deadline, often you can 
actually extend that deadline. It's often negotiable. '(15) 

Paradoxically, the deadlines appeared to be less important in the assessment of 
enough information. The deadline was there and it had to be met, but sometimes, 
a less-that-complete product was submitted, a product that did not have enough 
information: 'sometimes the briefings we send over aren't very good' (3); 'you find 
something a few days later and you think, 'gee, it would have been nice to have 
known about that" (3) 

And participants expressed concern that 'we were going to have to cut corners in 
terms of quality in order to get it done on time' (12). So while the deadline is part 
of nature of the business, whether or not the product contains enough information 
may be assessed independently of the deadline. 

Context of tasks and information seeking 

Participants were conscious of the dynamic and complex environment in which 
they operated: 'the political climate... that can change from hour to hour' (9) and 
'but the mood has changed now' (3). Collaboration appeared as a consistent theme 
in scoping both the task and the information seeking. In some instances, this was 
predetermined, with several participants involved in projects that crossed 
jurisdictions, increasing the number of stakeholders who held differing views 
about the purpose and nature of the task and adding to the complexity of what 
information was needed. The involvement of these multiple players resulted in 
experiences of information seeking that were 'coloured a lot by the processes that 
were going on around that [collaboration]' (12). However, even when 
participants had been asked to complete the task alone, they worked 
collaboratively, brainstorming with colleagues to create an initial framework for 
the task and relying on advice from colleagues and supervisors: 'I can't do this by 
myself (21), and 'grabbing anyone around', to use as a 'sounding board' (21). 

The tasks participants were dealing with were often vague and un-structured. So 
often, 'we didn't know basically what the scope of the work that needed to be 
done'{ sic) (12). Both the tasks and the related information seeking was difficult 
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because 'we're second guessing our Minister, the government has a public 
position that we don't support the initiative, but doesn't want to always be 
negative' (3). Further, goals were often unclear. Thus, 7 was feeling a little bit 
unsure about exactly what is was I was being set to do' { 15) and sometimes, 'there 
was miscommunication about what was needed' (18) and it was difficult to 
ascertain 'what counts - and what doesn 't' { 18). 

Adding to the uncertainty, participants often found themselves working in 
unfamiliar fields: 'it can be a topic that I personally know nothing about' (3). And 
even when they had knowledge of the broad field, the first important step was the 
scoping work needed 'to get the big picture' (21). Another characteristic of 
participants' experiences was the iterative nature of both task and information 
seeking: 'you can still, you can always reshape that, you can always shape that 
target' (12). 

Within this dynamic, complex and fluid environment, how did policy workers 
assess they had enough information? How did they make the decision to stop 
seeking more information? 

Frameworks against which to assess enough information 

One striking theme to emerge from the data was the need for a framework against 
which to make the assessment of enough information. At the outset, often 'you 
don't have any reference points' (21) and rarely was a framework provided or 
suggested by the supervisor requesting the task: 'there was not a lot of guidance' 
(15). Participants' understanding of the purpose of the task provided some 
guidance to them: 'It always depends on what's driving the collection of 
information' (12). Also, the organization's goals provided clues when no explicit 
framework was provided. Tasks were approached with a 'general understanding' 
(21) of the agency's organizational view of policy work helping to sort through 
what would be important and what not. 

But once that framework was established, the task became easier: 'There's a 
certain calming effect of actually getting the structure of whatever it is... and 
actually filling in the bits you can do already' (3); 7 had my framework [so it 
was] did I have enough information to write that bit of it, and could I substantiate 
why I put those bits in?' { 15). 

As participants worked on gathering information and working it into a draft 
document, they 'start[ed] to anticipate how it's going to be received... how they're 
going to react to it' (6). Towards the end of the task, when information had been 
gathered and an initial draft prepared, participants often used colleagues, 
supervisors and even formal reference groups as a feedback mechanism and, in 
some instances, relied on senior people to identify any gaps in their own 
assessment of enough information. Thus, participants 'gave it to someone else to 
read' to highlight gaps and 'if they change it, that's good in a way because... you 
know you're getting an idea for next time' (3). 
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While establishing a framework is a necessary first step and an essential part of 
the assessment of enough information, that framework is also fluid and 
changeable: 'your actual information needs have evolved throughout that whole 
experience' (6); 7 don't think you ever know [that you have enough]... it's a fluid 
thing' (9). 

Much in this analysis of experiences of the closing stages in information seeking 
activity supported existing research findings. For example, time as a constraint 
was a major theme and the tactic of drawing in colleagues as sounding boards 
when putting the written product together was reported by Foster ( 2004 ) and the 
creation and use of a framework against which to assess enough information 
echoed the concept of solving a puzzle ( Kuhlthau 2004 1 and integrating new 
information into existing knowledge ( Cole 1997 1. 

However, additional findings discussed below further extend our understanding of 
judgement and decision making during information seeking. Policy workers did 
not appear to experience the decision to stop seeking more information as a 
simple choice between two alternatives. Further, it was far from clear which, if 
any, stopping rules were being used. 

Insights into judgement and decision making during information 
seeking 

The experience of emerging empirical findings apparently at odds with the 
economic and mathematical models of normative decision theory, paralleled that 
of researchers into naturalistic decision making, a field of decision theory 
research developed during the 1990s. Naturalistic decision making research arose 
out of dissatisfaction with the theories of both optimal and behavioural decision 
making as explanations for how people make decisions ( Klein 1998 ). In the 
following section, key features only of the research approach are described; a full 
discussion of naturalistic decision making can be found in Montgomerery, et al. 
( 2005 ) and Zsambok and Klein 1 1997 1. 

Naturalistic decision making researchers investigate and describe how 
experienced people make decisions under constraints that are characteristic of 
real-world problem solving and decision making. Rather than conceptualising 
decision making as a single choice between two available options, naturalistic 
decision making researchers found that expert decision makers experience 
decision making as a process, and have increasingly focused on the role of 
judgement in decision making ( Kerstholt and Avton 2001 ). Characteristics of 
decision making in real world settings include contextual factors such as: 

• ill-structured problems; 

• uncertain and dynamic environments; 

• shifting, ill-defined or competing goals; 

• action, feedback loops; 

• time stress; 
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• high stakes; 

• multiple players; and 

• organizational goals and norms ( Zsambok and Klein 1997 : 5). 

Similarities were apparent between these characteristics and the experiences 
described by the policy workers participating in the research, similarities such as 
the complex changing environment and the time stress. A summary of these 
similarities appears in Table 1. 


Factors 

Descriptions of context by policy workers 

Ill-structured 

problems 

'we didn't know basically what the scope of the work 
that needed to be done' (sic) 

Uncertain and 

dynamic 

environments 

'the political climate... that can change from hour to 
hour'-, 'endless briefs, some of which weren't used 
because events were moving so fast' 

Shifting, ill- 
defined or 
competing 
goals 

'I was feeling a little bit unsure about exactly what is 
was I was being set to do' it was difficult to ascertain 
'what counts - and what doesn't' 

Action, 

feedback 

loops 

I 'start[ed] to anticipate how it's going to be 
received... how they're going to react to it'; 'if they 
change it, that's good in a way because... you know 
you're getting an idea for next time' 

Time stress 

The work is 'nerve-wracking [because] the timelines 
are so short now' 

High stakes 

'It's about really fundamental things'; 'It's a bit 
terrifying sometimes' 

Multiple 

players 

7 can't do this by myself, 'grabbing anyone around', 
to use 'as a sounding board' 

Organisational 
goals and 
norms 

7 suppose you do get to understand what's expected' 


Table 1: Factors characteristic of naturalistic decision making 

In these volatile contexts, how do the experts studied by naturalistic decision 
making researchers make decisions? They are continually assessing and updating 
their understanding of a situation as part of an iterative process towards 
determining appropriate action f Meso. et al. 2002 L By constructing and drawing 
on mental models of the situation, experts continually assess if that model is 
workable through a process of mental simulation, checking the plausibility of the 
model as the situation develops and changes around them and taking appropriate 
action ( Lipshitz. et al. 2001 ). The experts do not evaluate and choose between a 
range of options before making a decision but rather take action on the first 
feasible model that presents itself ( Lipshitz. et al. 2001 ). While this behaviour has 
parallels with satisficing behaviour, it does not appear to represent the classical 
definition of decision: a choice between two options. 

Parallels were apparent with the experiences reported by the research participants. 


http://www.infomiationr.net/ir/l l-4/paper266.html[6/21/2016 4:21:09 PM] 

















What defines 'enough' information? How policy workers make judgements and decisions during information seeking: preliminary results from an exploratory study 


Policy workers reported an essential first step was scoping or assessing the nature 
of the work task ahead of them: 7 didn't know what the parameters was that I was 
working in (sic)' (27). This scoping helped them develop a framework which they 
felt was necessary both for getting started on the task itself and as a guide to the 
information needed. This framework can be seen as the mental model identified 
by naturalistic decision making researchers. The framework is fluid and will 
change over time as feedback from supervisors or colleagues is received and 
incorporated: 'we were working so fast and the goalposts were moving at such a 
rapid rate' (27), a process of continuing assessment and updating of their 
understanding of what was required of them. Additional inputs to this 
understanding of the situation come from their understanding of organizational 
norms: 7 suppose you do get to understand what's expected' (15). Their 
assessment of enough information was carried out by answering the question: 
Enough for what? with what being delineated by the framework they had 
developed and modified. 

Conclusion 

Research findings support the criticality of developing a structure for a task 
( Vaklcari 2001 ) and a focus for the associated information seeking. Further, the 
findings suggest that this structure is also important during the closing stages of 
information seeking by providing that framework, ft is against this framework 
that assessments of enough information are made. 

Importantly, the similarities identified between the experiences of policy workers 
and the findings reported by naturalistic decision making researchers suggest their 
approach and the emerging theory has much to offer information seeking in 
context researchers. As noted earlier, where research has drawn on decision 
theory to understand judgement and decision making during information seeking, 
investigators have used classical or behavioural decision theory. However, 
drawing on these schools alone may limit our understanding of what is happening 
during search closure. For example, while much of the earlier research into 
stopping focused on modelling one-off sequential searches, with a choice to be 
made between two options, empirical findings suggest that this is not how people 
experience judgement and decision making while information seeking in real- 
world settings. 

The naturalistic decision making approach offers a different perspective on 
human judgement and decision making. Although not without limitations, for 
example, the role of affect in naturalistic decision making to date remains under¬ 
researched, this perspective concentrates attention on judgement and decision 
making as a multi-stage and fluid process rather than a linear and one-off choice. 
The importance and nature of the contextual factors which shape the processes 
involved are also emphas 

Findings from the study reported here suggest nonetheless that this emerging field 
can expand our understanding of how judgements and decisions are made when 
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seeking information in dynamic and complex environments, such as the public 
sector workplace. Analysis of the study data continues and it is expected that 
findings will enrich our knowledge of an important judgement made during 
information seeking. 

Notes 

1. After Sonnenwald (1998), the fundamental concepts of context and situation 
are seen in an interdependent relationship, with environment, the totality of 
surrounding factors, not all of which may be contextual factors related to 
particular situations. 

2. The numbers in parentheses (8) refer to the number of the interview from 
which the quotation is taken. 
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